Model Measures of Error

Measuring the quality of fit of a regression model is an important step in data modeling. There are several commonly used metrics used to quantify how well a model explains the data. By measuring the quality of fit we can select the model that makes the most accurate predictions on unseen data. Common metrics used to measure model performance are:

• Mean Absolute Error (MAE)

The Mean Absolute Error is a measure error magnitude. The sine of the error does not matter because MAE uses the absolute value. Small MAE values, “lower magnitude” indicate better model fit. MAE is calculated (1) by averaging the absolute difference between the observed \((y_i)\) and predicted \(\hat{f}(x_i)\) values. Where:

• \(n\) is the number of observations,
• \(\hat{f}(x_i)\) is the prediction that the regression function \(\hat{f}\) gives for the ith observation,
• \(y_i\) is the observed value.

\[ \text{MAE} = \frac{1}{n} \sum_{i=1}^n |y_i - \hat{f}(x_i)| \tag{1} \]

• Root Mean Squared Error (RMSE)

The Root Mean Squared Error (2) is a measure of error magnitude also. Like MAE, smaller RMSE values indicate better model fit. In this method the square error \((y_i - \hat{f}(x_i))^2\) values are used. Squaring the error give more weight to the larger ones. In contrast with the MAE that uses the absolute error \(|y_i - \hat{f}(x_i)|\) values, all errors are weighted equally regardless of size. Taking the square root returns the error to the same units as the response variable, making it easier to interpret.

\[ \text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{f}(x_i))^2} \tag{2} \]

• R-squared (\(R^2\))

R-squared is the percent of the variance in the response variable that is explained by the predictor variable(s). Unlike MAR and RMSE, \(R^2\) values range from 0 to 1 and the higher the value, the better the fit. An \(R^2\) value of 0.75 indicates that 75% of the variance in the response variable can be explained by the predictor variable(s). The \(R^2\) equation (3) is composed of two key parts, the Total Sum of Squares (\(SS_{tot}\)) and the Residual Sum of Squares (\(SS_{res}\)).

\[ \text{R}^2 = \frac{SS_{tot}-SS_{res}}{SS_{tot}} = 1 - \frac{SS_{res}}{SS_{tot}} = 1 - \frac{\sum_{i=1}^{n}(y_i - \hat{f}(x_i))^2}{\sum_{i=1}^{n}(y_i-\bar{f}(x_i))^2} \tag{3} \]

(James et al. 2013), (Hawkins, Basak, and Mills 2003), (Helsel and Hirsch 1993)

Leave One Out Cross-validations (LOOCV)

\[ CV_{(n)} = \frac{1}{n}\sum_{i=1}^{n} \text{Measuer of Errori}_i \tag{5} \]

Process: The steps for LOOCV are almost identical to k-fold cross-validation. The only difference is that in K-fold, K must be less than the number of observations (n). In LOOCV, K = n, so when you split the data into testing and training data, the first testing fold is one of the observations and the training data would be every other observation ?@fig-LOOCV. In this way, every observation is tested against every other observation and the process would be repeated n times (James et al. 2013).

Data extraction, transformation and visulation

(I-C Yeh 1998) modeled compression strength of high performance concrete (HPC) at various ages and made with different ratios of components ?@tbl-data. The data used for their study was made publicly available and can be downloaded UCI Machine Learning Repository (I-Cheng Yeh 2007).

LightGBM Model

Modle Comparison K-Fold Plot

Modle Comparison LOOCV Plot

Modle Comparison Nested CV Plot

LightGBM (Light Gradient Boosting Machine)

• Description: A gradient boosting framework that uses tree-based learning algorithms.
• Pros: High efficiency, fast training, and capable of handling large datasets.
• Cons: Requires careful tuning of parameters.

Predictive Performance Comparison

• LightGBM outperformed traditional models and cross-validation techniques.
• Lower prediction errors and more reliable performance metrics.
• Demonstrated strong generalization capabilities.

Computational Efficiency

• LightGBM: Fast training and efficient computation.
• Nested Cross-Validation: Excellent performance but computationally intensive.
• Efficiency crucial for real-world applications with limited resources.

Conclution conclusion

• Cross-validation techniques and LightGBM effectively reduce overfitting and enhance model accuracy.
• LightGBM offers superior accuracy and efficiency.
• Identified key predictors for accurate model development.
• Robust framework for model evaluation, improving decision-making in concrete design and construction.

Future Research

• Further refinement of these techniques to improve predictive accuracy.
• Exploration of additional advanced models.
• Application in various engineering contexts to enhance model reliability and performance.